Image generating apparatus, image generating method, and recording medium having the program stored thereon

ABSTRACT

An objective is recognition of the travel direction of an object captured in an image frame. An image generating apparatus includes: a receiver circuit that receives a single exposure frame created by a single exposure and a multiple exposure frame created by multiple exposures after the single exposure frame, the single exposure frame and the multiple exposure frame being output from an image sensor; and a control circuit that specifies a starting point of a motion track of an object in the multiple exposure frame in accordance with a position of the object in the single exposure frame and a position of the same object in the multiple exposure frame.

CROSS REFERENCE TO RELATED APPLICATIONS

The disclosure of Japanese Patent Application No. 2017-056565 filed onMar. 22, 2017 including the specification, drawings and abstract isincorporated herein by reference in its entirety.

TECHNICAL FIELD

The present disclosure relates to an image generating apparatus, imagegenerating method, a program, and a recording medium having the programstored thereon.

BACKGROUND ART

A technique is known in which a moving object is serially imaged bymultiple exposures in such a manner that the same object in differentpositions is captured in a single image to express the motion of theobject.

CITATION LIST Patent Literature

PTL 1

Japanese Patent Application Laid-Open No. 2014-23067

SUMMARY OF INVENTION Technical Problem

However, the travel direction of the object cannot be recognized from asingle image made by serial imaging by multiple exposures. There is amethod for recognizing the travel direction of the object by varying theconcentration in such a manner that the sensitivity is graduallydecreased (or increased) in each timing of multiple exposures. In thiscase, it is difficult to recognize the object itself that is imaged inthe timing of exposure with decreased sensitivity.

For this reason, an object of the present disclosure is to provide animage generating apparatus, an image generating method, a program, and arecording medium having the program stored thereon, which allow anobject and the travel direction of the object to be recognized.

Solution to the Problem

An image generating apparatus according to one embodiment of the presentdisclosure includes: a receiver circuit that receives a single exposureframe created by a single exposure and a multiple exposure frame createdby multiple exposures after the single exposure frame, the singleexposure frame and the multiple exposure frame being output from animage sensor; and a control circuit that specifies a starting point of amotion track of an object in the multiple exposure frame in accordancewith a position of the object in the single exposure frame and aposition of the same object in the multiple exposure frame.

The aforementioned embodiment can be any one of a method, a program, anda non-transitory tangible recording medium storing a program.

Advantageous Effects of Invention

The present disclosure allows an object itself to be recognized and thetravel direction of the object to be recognized.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram of an example configuration of an imagingapparatus including an image generating apparatus according toEmbodiment 1;

FIG. 2 is a diagram showing examples of image frames output from animage sensor;

FIG. 3 is a flow chart of an example of the operation of controlsection;

FIG. 4 is a diagram showing a modification of image frames output froman image sensor; and

FIG. 5 is a block diagram of an example configuration of an imagingapparatus including an image generating apparatus according toEmbodiment 2.

DESCRIPTION OF EMBODIMENTS

An embodiment will now be described with reference to accompanyingdrawings.

It should he noted that when elements of the same type are distinguishedfrom each other for description, reference numerals, such as “Object200A” and “object 200B”, may he used, and when elements of the same typeare not distinguished from each other for description, only a commonnumber in the reference numeral, such as “object 200”, may be used.

In addition, in the embodiments below, the constituent elements(including constituent steps) are not necessarily essential unlessotherwise clearly stated or clearly regarded as being theoreticallyessential.

Embodiment 1

FIG. 1 is a block diagram of the configuration of an imaging apparatusincluding an image generating apparatus according to Embodiment 1. FIG.2 is a diagram showing image frames output from an image sensor.

Imaging apparatus 10 includes image sensor 12, receiver section 14, andcontrol section 16. It should be noted that functions related toreceiver section 14 and control section 16 may be implemented by imagevenerating apparatus 11 which is, for example, a microprocessor ordigital-signal processor. Alternatively, the functions may beimplemented by a program executed on a computer including at least amemory and a processor, or when the computer reads the recording mediumhaving the program stored thereon.

Image sensor 12 includes a plurality of pixels (a pixel array) includingphotoelectric conversion elements, and image frames are generated fromsignals obtained by photoelectric conversion of light incident on eachpixel and then output to receiver section 14.

Image sensor 12 is a complementary metal oxide semiconductor (CMOS)image sensor made using, for example, an organic thin film. It should benoted that the image sensor is not limited to this and may be anotherCMOS image sensor or a charge coupled device (CCD) image sensor.

Image sensor 12 has the global shutter function. With the global shutterfunction, the start time and end time of exposure are common to all thepixels in the pixel array. The global shutter function allows arelatively fast-moving object to be imaged without distortion. Exposuretime and the number of exposures for image sensor 12 can be changed foreach image frame.

An image frame created by a single exposure is referred to as “singleexposure frame 100”. In other words, single exposure frame 100 is animage frame created by releasing the global shutter once. In FIG. 2,object (another vehicle) 200 is moving from left to right in thedrawing. Single exposure frame 100 contains object 200A imaged by asingle exposure.

An image frame created by more than one exposures is referred to as“multiple exposure frame 120”. In other words, multiple exposure frame120 is an image frame created by releasing the global shutter more thanonce (through multiple exposures). Objects 20013, 200C, and 200D inmultiple exposure frame 120 and object 200.A in single exposure frame100 are identical object 200. Multiple exposure frame 120A containsobjects 200B, 200C, and 2001) captured in the respective timings ofthree exposures. Objects 20013, 200C, and 200D deviate little by little,resulting in multiple imaging. This is because object 200 moves duringrelease of the global shutter. In multiple exposure frame 120A in FIG.2, multiple backgrounds overlap because the vehicle from which the imageis taken (hereinafter, target vehicle) is also running forward.

Image sensor 12 generates single exposure frame 100 and multipleexposure frame 120 alternately and inputs them to receiver section 14.In other words, image sensor 12 repeats processing in which singleexposure frame 100 is generated, multiple exposure frame 120 (the nextframe) is generated, and single exposure frame 100 (the next frame) isgenerated again. Image sensor 12 may output single exposure frame 100and multiple exposure frame 120 at the same frame rate. Thus, aprocessing load on control section 16 can be reduced.

Receiver section 14 receives single exposure frame 100 output from imagesensor 12 and transmits it to control section 16. Receiver section 14receives multiple exposure frame 120 output after that single exposureframe 100 and transmits it to control section 16.

Control section 16 receives multiple exposure frame 120 and singleexposure frame 100 from receiver section 14. Subsequently, controlsection 16 specifies a starting point of the motion track of object 200in multiple exposure frame 120 in accordance with a position of object200 in single exposure frame 100 and a position of object 200 inmultiple exposure frame 120. The details will be explained below withreference to FIG.

As shown in FIG. 2, multiple exposure frame 120A contains multipleobjects 200B, 200C, and 200D. Control section 16 specifies the motiontrack of object 200 in accordance with a positional relationship betweenmultiple objects 200B, 200C, and 200D contained in multiple exposureframe 120A. However, the starting point of the motion track of object200 cannot be clear only from multiple exposure frame 120A. In otherwords, the travel direction of object 200 cannot be clear.

For this reason, control section 16 specifies the starting point of themotion track of object 200A in this multiple exposure frame 120 by usingthe previous frame of multiple exposure frame 120, i.e., single exposureframe 100.

For example, control section 16 specifies, among multiple objects 20013,200C, and 200D contained in multiple exposure frame 120A, position 210Bof object 20013 that is the closest to position 210A of object 200Acontained in single exposure frame 100, and determines that thisspecified position 21013 is the starting point of the motion track ofobject 200. This is because the time gap between the time when singleexposure frame 100 was created and the time when the next frame, i.e.,multiple exposure frame 120A was created is relatively short and thetravel distance of object 200 during the gap is therefore assumed to berelatively short.

It should be noted that position 210 of object 200 in the image framemay correspond to the central coordinates of the imaging region of thisobject 200. In addition, the motion track of object 200 may be adirection vector connecting the starting point with the coordinates ofeach object.

Control section 16 can issue at least one of the following controlcommands (or control signals) 160 (C1) and (C2) to image sensor 12 tocontrol the operation of image sensor 12.

(C1) Control command 160 for instruction to alternately generate singleexposure frame 100 and multiple exposure frame 120.

(C2) Control command 160 for instruction to use a common sensitivity ineach timing of exposure for multiple exposure frame 120. Using the samehigh sensitivity prevents images from being captured at low sensitivityand object 200 from being not recognized

FIG. 3 is a flow chart of processing in control section 16. Processingin control section 16 will now be explained with reference to FIGS. 2and 3.

Control section 16 receives, from receiver section 14, single exposureframe 100 and the next frame of this single exposure frame 100, i.e.,multiple exposure frame 120A (ST100).

Next, control section 16 specifies object 200A from single exposureframe 100. In addition, control section 16 specifies that position 210Aof object 200A in single exposure frame 100 (ST102). For example,control section 16 specifies object 200.A by a predetermined patternmatching method.

Control section 16 then specifies objects 20013, 200C, and 2001) whichare identical to object 200A specified from multiple exposure frame 120Ain ST102. Control section 16 also specifies positions 210B, 2100, and210D of these objects 200B, 200C, and 200D in multiple exposure frame120A (ST104).

Control section 16 then specifies, among positions :210B, 210C, and 210Dof the same object, i.e., objects 200B, 200C, and 200D in multipleexposure frame 120 specified in ST104, position 210B as the startingpoint of the motion track of object 200 in accordance with position 210Aof object 200A specified in ST102 (ST106). Here, among positions 21013,210C, and 210D, position 21013 that is the closest to position 210A isspecified as the starting point.

Control section 16 then generates the motion track of object 200(direction vector following positions 210B to 210D in the orderpresented) in accordance with a positional relationship between startingpoint position 210B of object 200 specified in ST106 and each ofpositions 210B, 210C, and 210D of objects 200B, 200C, and 200D specifiedin ST104 (ST108).

Control section 16 then outputs information 140 related to the motiontrack of object 200 generated in ST108 to a predetermined apparatus inthe subsequent stage (ST110). The predetermined apparatus in thesubsequent stage is, for example, an electronic control section (ECU)mounted on a vehicle for controlling automated driving or driveassistance. It should he noted that control section 16 may output singleexposure frame 100 and multiple exposure frame 120 to the predeterminedapparatus in the subsequent stage together with or instead ofinformation 140 related to the motion track. Control section 16 mayoutput single exposure frame 100 and multiple exposure frame 120 at thesame frame rate. Thus, a processing load on the predetermined apparatusin the subsequent stage can be reduced.

Control section 16 repeats the aforementioned processing of ST100 toST110.

When control section 16 specifies multiple different objects 200 inST102, each object 200 is subjected to processing of ST104 to ST110.Accordingly, the motion track of each of multiple different objects 200can be generated.

It should be noted that control section 16 grants an ID to each object200 specified in ST102 and, upon specification of the same object 200 inthe second or later repeated processing, the same ID is granted to thesame object 200. Accordingly, the motion of the same object 200 can becontinuously tracked.

Through the aforementioned processing, the starting point of the motiontrack of object 200 can be recognized. In other words, the trawldirection of object 200 can be recognized. Further, as imaging isenabled with relatively high sensitivity compared with the case wherethe travel direction is specified by changing sensitivity, the accuracyof object recognition is improved.

(Modification)

FIG. 4 is a diagram showing modifications of image frames output fromimage sensor 12. With reference to FIG. 4, described below is the casewhere image sensor 12 generates single exposure frame 100 and repeatsprocessing in which more than one multiple exposure frames 120A and 120Bare generated.

Receiver section 14 transmits single exposure frame 100 and thefollowing more than one multiple exposure frames 120A and 120B tocontrol section 16.

As described above, control section 16 uses single exposure frame 100 tospecify starting point position 210B of object 200 from the next frameof that single exposure frame 100, i.e., multiple exposure frame 120A.

Control section 16 then generates the motion track of object 200starting from starting point position 21013 (direction vector followingpositions 210B to 210G in the order presented) in accordance withpositions 210B to 210G of objects 200B to 200G captured in more than onemultiple exposure frames 120A and 120B that follow single exposure frame100.

In this case, control section 16 can issue the following control command(or control signal) (C3) in addition to the aforementioned (C1) and (C2)to image sensor 12 to control the operation of image sensor 12,

(C3) Control command 160 for instruction to generate N (N is an integerof two or more) multiple exposure frames 120 after single exposure frame100.

Through the aforementioned processing, even a configuration in whichmore than one multiple exposure frames 120 follows one single exposureframe 100 allows the starting point of the motion track and traveldirection of object 200 to be recognized.

Embodiment 2

In Embodiment 2, described is a configuration in which each pixel in animage sensor includes a high-sensitivity cell and a highly-saturatedcell. It should be noted that the same component as that in FIG. 1 canbe denoted by the same reference numeral for omission of description.

FIG. 5 is a block diagram showing the configuration of imaging apparatus13 including image generating apparatus 31 according to Embodiment 2.

Image sensor 32 includes high-sensitivity cell 310 and highly-saturatedcell 312 in each pixel.

High-sensitivity cell 310 has relatively high sensitivity and relativelysmall charge storage capacity. Accordingly, high-sensitivity cell 310achieves imaging of, for example, traffic signs, traffic signals,pedestrians, or other vehicles even in dark places, e.g., at nighttimeand in a tunnel.

On the contrary, highly-saturated cell 312 has lower sensitivity andlarger charge storage capacity than high-sensitivity cell 310.Accordingly, highly-saturated cell 312 can mainly image light emitted byobjects in dark places, such as light from headlights or taillights ofother vehicles, traffic signals, and street lights.

High-sensitivity cell 310 performs imaging by a single exposure andoutputs high-sensitivity single exposure frame 320. Highly-saturatedcell 312 performs imaging by multiple exposures and outputshighly-saturated multiple exposure frame 322.

Control section 36 performs the same processing as in Embodiment 1 byusing high-sensitivity single exposure frame 320 and highly-saturatedmultiple exposure frame 322 received from image sensor 32, therebyspecifying the motion track of an object. For example, control section36 specifies the starting point of a moving object throughhigh-sensitivity single exposure frame 320 and specifies the track ofthe moving object through highly-saturated multiple exposure frame 322.Accordingly, the track of the moving object can be specified even ifsurroundings of the target vehicle are in a dark place.

In image sensor 32, exposure time and the number of exposures can bechanged for each of high-sensitivity cell 310 and highly-saturated cell312. Further, control section 36 may be able to set such exposure timeand the number of exposures through predetermined command 160.

Regarding image sensor 32, single exposure frame 320 fromhigh-sensitivity cell 310 and multiple exposure frame 322 fromhighly-saturated cell 312 may be output at the same frame rate. Thus, aprocessing load on control section 36 can be reduced.

Alternatively, highly-saturated cell 312 may output an image frame byalternately performing imaging by a single exposure and imaging bymultiple exposures. Accordingly, control section 36 can specify thestarting point of the moving object also from the image frame outputfrom highly-saturated cell 312.

In addition, control section 36 may output single exposure frame 320output from high-sensitivity cell 310 and multiple exposure frame 322output from highly-saturated cell 312 to the predetermined apparatus inthe subsequent stage together with or instead of information 140 relatedto the motion track. At this time, control section 36 may output singleexposure frame 320 and multiple exposure frame 322 at the same framerate. Thus, a processing load on the predetermined apparatus in thesubsequent stage can be reduced.

The above embodiments may be mere examples for explaining the presentdisclosure, and the scope of the present disclosure may not be limitedto only these embodiments. The present disclosure can be implemented bythose skilled in the art in any other ways without departing from thesummary of the present disclosure.

INDUSTRIAL APPLICABILITY

An image generating apparatus, an image generating method, a program,and a recording medium related to the present disclosure allow thetravel directions of moving objects to be recognized, and are suitablefor use in imaging apparatuses, car-mounted equipment, and the like.

REFERENCE SIGNS LIST

-   10 Imaging apparatus-   11 Image generating apparatus-   12 Image sensor-   14 Receiver section-   16 Control section-   100 Single exposure frame-   120 Multiple exposure frame.-   200 Object

1. An image generating apparatus comprising: a receiver circuit thatreceives a single exposure frame created by a single exposure and amultiple exposure frame created by multiple exposures after the singleexposure frame, the single exposure frame and the multiple exposureframe being output from an image sensor; and a control circuit thatspecifies a starting point of a motion track of an object in themultiple exposure frame in accordance with a position of the object inthe single exposure frame and a position of the same object in themultiple exposure frame.
 2. The image generating apparatus according toclaim 1, wherein a common sensitivity is applied to each timing ofexposure for the multiple exposure frame.
 3. The image generatingapparatus according to claim 1, wherein the image sensor includes ahigh-sensitivity cell and a highly-saturated cell that has lowersensitivity and larger charge storage capacity than the high-sensitivitycell, and the single exposure frame and the multiple exposure frame arecreated by the highly-saturated cell.
 4. The image generating apparatusaccording to claim 1, wherein the single exposure frame and the multipleexposure frame are created by a global shutter function of the imagesensor.
 5. The image generating apparatus according to claim 1, whereina common frame rate is applied to the single exposure frame and the mexposure frame.
 6. An image generating method comprising: inputting,through an image sensor, a single exposure frame created by a singleexposure and a multiple exposure frame created by multiple exposuresafter the single exposure frame; and specifying a starting point of amotion track of an object in the multiple exposure frame in accordancewith a position of the object in the single exposure frame and aposition of the same object in the multiple exposure frame.
 7. Anon-transitory recording medium storing a program that causes a computerto: receive a single exposure frame created by a single exposure and amultiple exposure frame created by multiple exposures after the singleexposure frame, the single exposure frame and the multiple exposureframe being output from an image sensor; and specify a starting point ofa motion track of an object in the multiple exposure frame in accordancewith a position of the object in the single exposure frame and aposition of the same object in the multiple exposure frame.